Skip to content

Persist bounded lead facts with checkpoint replay and stall incidents (#629) - #684

Merged
oxa-orchestrator[bot] merged 12 commits into
mainfrom
agent/629-lead-facts
Nov 17, 2026
Merged

oxa-orchestrator[bot] merged 12 commits into
mainfrom
agent/629-lead-facts

Conversation

@n0rpan

@n0rpan n0rpan commented Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator

The lead needs current facts without reading private runner state or interpreting logs. This adds the L1 events, situation and incidents modules in oxa-work, consuming #669's shared types. The runner retains task state, transition emission, checkpoint adapters and CLI/action wiring.

Tier T3 for protected durable files, rotation and recovery. The exact manifest hunk proposes only existing workspace oxa-proto/libc edges and their lock edges (shared types and no-follow I/O); no dependency version is added. Manifest-owner review is requested as part of this PR.

  • Facts::initialize explicitly creates a fresh store; open never converts missing/corrupt/partial data into an empty view. Supply exactly the protected repository root's facts/ child, outside writable workspace/Git binds and separate from task receipts.
  • append(checkpoint, at_ms, fact) validates the projection, saves a pending observation, appends/syncs its journal line, commits the source cursor and atomically replaces the situation. Recovery handles a reserved torn line, interrupted rotation, lost acknowledgement and lagging projection. Conflicting facts, stale revisions, unrecoverable gaps and sequence exhaustion are errors.
  • Runner confirmed repository-wide revisions, fixed source runner, serialized checkpoint/callback/ack and no replay of acknowledged older tasks. Revision gaps are accepted; event numbers stay contiguous. The bounded emitter table retains no per-task tombstones or lifetime task cap.
  • Situation reports running location/closed task step, acceptance/publication, landed PRs, blocks, asks and incidents. Only PrMerged means landed. Active rows are never silently dropped. Bounded terminal/incident history marks truncation explicitly; new heads cannot clear blocks/asks.
  • events_after returns bounded exclusive-cursor pages with explicit gaps; two 256 KiB journals rotate without resetting sequence. The incident list is a projection of that same journal.
  • Stall detection validates bounded status bytes, expected saved-session ID/PID, timeout and conservative deadline. It returns an incident description and never wraps/kills/polls; runner owns the decided wall-time policy.
  • Private 0700/0600 storage, no-follow regular-file/ownership checks, file/directory sync and serialized appends. No model/network/daemon, private State access or raw prompt/diagnostic data.

Includes the requested TaskTransition/TaskStep/StepReason extension and explicit Unblocked observations. API, recovery and adapter contract.

Validation on f8753be, after a normal merge of main 30a2d21:

  • 3,860 complete workspace tests passed, mandatory bwrap, inventory/report guard, zero runtime skips; only the three existing opt-in measurement/dump ignores.
  • Doctests, strict workspace/all-target clippy, formatting, four unchanged scans and changed-document links passed.
  • Real storage tests cover all commit crash boundaries, lost acknowledgements, torn lines, rotation, global revision gaps, concurrent sources, capacity, exact unblocking, private/no-follow files and an unchanged runner-state sentinel. Stall tests cover identity, deadlines, disabled/finished states and invalid data.
  • All three confirmed review findings are fixed/answered after committed failing-first regressions: history byte-cap eviction, containing-directory sync and matching-pending-line journal sync before acknowledgement. Native syscall fault injection proves EIO leaves pending metadata unchanged and a clean retry recovers once, for both retained journals.
  • The merge retains main's canonical collision-safe gateway fixture and its private/independent-root regression, superseding the earlier allocator from this PR. A rebuilt native consumer still keeps two live real fixtures under one frozen clock tick. The shared fixture file is unchanged relative to main.
  • Current head's full ready PR run 43899182063 and named gate are green, including both mandatory-bwrap workspace test shards. Automated review and the current-head independent gate read are clear.
  • The merged runner's Running wire type requires an explicit step. The old shared evening fixture failed all three consumers before ab952f4; four running values now explicitly say preparing. Three focused fixture tests and the entire workspace pass. Event/snapshot counts, sequences and the intentional torn tail are preserved.

On the unchanged storage implementation before this main refresh, a native public-API probe (isolated synthetic files, stripped Rust 1.99.1) measured: 4,000 appends plus situation/page/incident reads; one thread throughout. RSS was 1,188 KiB before initialization, 1,388 after, 1,936 after 4,000/read; observed peak 2,480 KiB, cumulative CPU 31.13s. Journals ended at 118,721 / 261,958 bytes, metadata at 290 and situation at 238; sequence=4,000, one emitter, no pending observation, explicit retention gap. The normal append path avoids duplicate complete replay. This is synthetic filesystem work, not model/provider performance.

Private composition with runner #69211dc511 also passed real task/check/save through the public callback: lost-ack recovery, nine contiguous facts, one worker completion, saved session and a finished retry with no new work. Independent proof now drives immutable runner preview 57a5c0c through literal work situation (text/JSON) and work events --after N while a real child waits on a loopback model request. It shows the live task/host/branch/worktree/saved session at cursor4, clears running on completion at cursor9 without claiming acceptance/landing, and a cached retry makes zero requests/events. A planning refusal becomes an explicit Unblocked after the missing Git ref is restored with unchanged task identity. The same immutable preview passes all 3,608 workspace tests, inventory/report guard with zero runtime skips, doctests, strict workspace/all-target clippy, fmt, unchanged scans and document links. Earlier crash fixtures now fail at real checkpoint callbacks rather than rewinding finished state. No task-state/history workaround or invariant relaxation was applied. This is preview integration proof; the runner's eventual main-based PR needs its own current-head checks.

Default CLI footprint (module not wired into CLI yet): same build path/toolchain, stripped release versus main 32297f7; 20 interleaved ABBA saved-session RPC/mock samples with confirmed active streaming.

Measure Main 32297f7 L1 4cf042b
Binary bytes 14,879,608 14,857,144 (-22,464)
Idle RSS median (range), KiB 9,576 (8,996–9,820) 9,582 (9,284–9,716)
Streaming RSS median (range), KiB 9,664 (9,084–9,908) 9,734 (9,436–9,868)
Idle / streaming anonymous RSS medians, KiB 1,116 / 1,204 1,116 / 1,204
Idle / streaming threads 4 / 4 4 / 4
Worker first-output median (range), ms 5.593 (4.412–13.221) 5.997 (4.109–8.679)
Paired direct-request TTFB median (range), ms 2.898 (0.753–11.381) 5.289 (1.142–9.727)
Worker minus direct TTFB median (range), ms 2.703 (-5.728–4.545) 1.635 (-3.588–3.996)

The default path creates no facts store or new thread. Anonymous medians are identical; the +6 KiB idle / +70 KiB streaming total-RSS medians come from the file-backed working set and link layout, within overlapping ranges. Binary size differences are not a claimed optimization. Interleaved timings are noisy (including negative paired differences), so no speed claim is made. Actual store-use cost is measured separately above.

This is L1's library layer, not all of #629. Runner adoption and real work CLI proofs remain in the active goal; L2 standing follows that boundary. The author does not queue or merge.

@n0rpan
n0rpan marked this pull request as ready for review November 16, 2026 11:36
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-16T16:49:13.328497Z 4cf042b Draft marked ready
🔒 Security Review ✅ Completed 2026-11-17T13:32:19.245966Z 4cf042b Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f765d3b06a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread oxa/oxa-work/src/situation.rs Outdated
}
snapshot.as_of_seq = event.seq;
snapshot.at_ms = event.at_ms;
snapshot.to_json().map_err(|error| error.to_string())?;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Trim retained history before rejecting the snapshot byte limit

When the situation reaches the 64 KiB byte limit before reaching 128 rows, serialization rejects the observation without attempting to evict retained landed/incident history. For example, 95 valid asks with 512-character questions plus 16 incidents with bounded identifiers exceed 64 KiB, while those same active asks without incident history occupy approximately 56 KiB. Because append must succeed before the runner's next action, disposable history can unnecessarily stop progress. Apply history eviction for the byte limit as well as the row limit, setting history_truncated, and reject only when the active projection itself cannot fit.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 65f463c (failing-first regression f0da0e0). Projection now discards only landed/incident history until both row and serialized-byte limits fit, marking history_truncated; active rows are never removed. The real-store regression reproduces the original failure with 95 long asks plus 16 incidents, then proves asks survive, reopen agrees, and active-only overflow refuses without advancing the sequence or changing the prior snapshot. All 18 focused storage/stall tests pass; full workspace verification is running.

Comment on lines +47 to +50
io(
fs::DirBuilder::new().mode(0o700).create(path),
"cannot create facts root",
)?;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Sync the parent directory after creating the facts root

When initialization creates the facts directory, its entry in the containing repository directory is never synced. Subsequent Root::sync calls sync only the new facts directory, which does not guarantee durability of its entry in its parent. A power loss shortly after successful initialization and acknowledged appends can therefore lose the entire store despite its files having been synced. Sync the containing directory after creating the root before reporting successful initialization.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 65f463c (failing-first regression f0da0e0). Explicit initialization syncs the containing directory before any successful initialization, including retries where mkdir already happened. The existing child-directory sync remains on file publication. A real permission regression makes the parent writable/searchable but unreadable: the old implementation initialized successfully, while the fixed implementation refuses the failed parent-directory open/sync. It runs in a subprocess and drops root privileges when necessary; no test skip. All 18 focused storage/stall tests pass; full workspace verification is running.

@n0rpan

n0rpan commented Nov 16, 2026

Copy link
Copy Markdown
Collaborator Author

First-pass review by oxa → oxc → openai/gpt-6.1-sol at head f765d3b (O-241; the orchestrator adjudicates and enqueues).

Codex findings

  • CONFIRMED — P2, oxa/oxa-work/src/situation.rs:130: Byte-limit validation still rejects snapshots without evicting landed/incident history; retained history can stop appends even when active facts alone fit.
  • CONFIRMED — P1, oxa/oxa-work/src/facts_io.rs:47–50: Creating the facts root never syncs its parent directory; power loss can lose the entire acknowledged store despite subsequent file/root-directory syncs.
    TIER: T2
    VERDICT: REVISE

🤖 Generated with Claude Code (oxa review lane)

@n0rpan

n0rpan commented Nov 16, 2026

Copy link
Copy Markdown
Collaborator Author

gatekeeper: Independent read of f765d3b.

Findings: no P1–P3 findings in this library scope.

Checked the entire diff, shared fact codecs and validation, journal/snapshot recovery ordering, rotation boundaries, checkpoint deduplication, bounded source/active/history tables, cross-handle lock acquisition, and private file operations. Pending state precedes paid-for durability claims; the journal is synced before the source cursor commits, and lagging projections recover before another append. Tests cover torn writes, post-append and post-commit recovery, interrupted rotation, lost acknowledgements, concurrent emitters, retention gaps, source/active capacity, poisoned leaves and explicit unblocking. Worker completion remains distinct from acceptance/publication/merge. Stall detection validates invocation identity and returns an observation without killing or polling. Fixed storage bounds and no background runtime are proportionate; the reported store-use CPU cost is visible rather than hidden behind idle CLI measurements. Named gate is green; no builds run. Runner adoption and real CLI boundary proof remain follow-up work.

Touches oxa-work: the orchestrator must still perform its own independent read after this comment.

@n0rpan
n0rpan marked this pull request as ready for review November 16, 2026 12:06

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 65f463c876

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +293 to +297
} else if journal.events.last() != Some(&event) || journal.torn_at.is_some() {
return Err("pending fact disagrees with the journal".into());
}
state.committed(&event)?;
self.root.atomic(STATE, &state.bytes()?)?;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Sync an already-written pending event before committing recovery

If the writer exits after writing the complete journal line but before Root::append finishes sync_all (or that sync returns an error), recovery can see the matching line in the page cache and take this branch without syncing it. Recovery then durably clears pending and acknowledges the observation, although the journal data is not guaranteed durable; syncing the metadata replacement and directory does not sync the journal's contents. A subsequent power loss can leave committed metadata ahead of the journal, making the store fail to reopen. Sync the existing matching journal line before committing the recovered state, just as the replay-and-append path does.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed and fixed in 667f15a (committed red regression d325ab2). The matching complete-line recovery branch now opens and syncs the actual retained journal before committing the source cursor/clearing pending. A permission regression failed before/pass after; all 19 focused tests pass. A separate native strace fault probe proves the durability boundary: the old head acknowledged with no journal fsync; the fix receives an injected EIO from that journal's fsync, leaves the complete pending metadata byte-for-byte unchanged, and then recovers exactly one event on an uninjected retry. This was also proved when the matching event is in the previous journal with an empty current journal. Full workspace/current-head gate are running; PR stays draft meanwhile.

@n0rpan

n0rpan commented Nov 16, 2026

Copy link
Copy Markdown
Collaborator Author

First-pass review by oxa → oxc → openai/gpt-6.1-sol at head 65f463c (O-241; the orchestrator adjudicates and enqueues).

Codex findings

  • FIXED — oxa/oxa-work/src/situation.rs:130: serialization failure now evicts retained history before rejecting an oversized active projection.
  • FIXED — oxa/oxa-work/src/facts_io.rs:59: initialization now syncs the containing directory, including on retries.
  • CONFIRMED — P1, oxa/oxa-work/src/events.rs:296–300: recovery accepts an already-written matching pending line without syncing the journal before durably clearing pending. If the original writer exited before journal sync, a subsequent power loss can lose the acknowledged event and leave metadata ahead of the journal, preventing reopening.
    TIER: T2
    VERDICT: REVISE

🤖 Generated with Claude Code (oxa review lane)

@n0rpan

n0rpan commented Nov 16, 2026

Copy link
Copy Markdown
Collaborator Author

gatekeeper: Independent reread of 65f463c.

Findings: no P1–P3 findings at this head.

Rechecked the complete effective diff against the prior reviewed head and reread the affected projection/initialization paths. Disposable history now shrinks for the encoded-byte cap as well as row count; active-only overflow refuses before reserving a sequence, with unchanged snapshot asserted. Initialization syncs the containing directory even on retry, and sync errors propagate. The new tests exercise actual capacity and permission failures. The remainder of the full diff is unchanged from my earlier journal/recovery, concurrency, private-file, stall and resource review. Named gate is green. No builds or credential files read.

Touches oxa-work: the orchestrator must still perform its own independent read after this comment.

@n0rpan
n0rpan marked this pull request as draft November 16, 2026 12:13
@n0rpan
n0rpan marked this pull request as ready for review November 16, 2026 12:22
@n0rpan

n0rpan commented Nov 16, 2026

Copy link
Copy Markdown
Collaborator Author

gatekeeper: Independent reread of 667f15a.

Findings: no P1–P3 findings at this head.

Rechecked the full diff against previously reviewed 65f463c and reread pending recovery ordering. A complete visible pending line is now synced again before committing metadata/acknowledging it, using the current or retained journal as appropriate. The real permission-failure regression proves recovery leaves state/projection unchanged when that sync cannot be established and succeeds after access is restored. The prior bounds, history trimming, private-file, serialization and no-background-runtime assessment remains applicable. Named gate is green. No builds or credential files read.

Touches oxa-work: the orchestrator must still perform its own independent read after this comment.

@n0rpan
n0rpan marked this pull request as draft November 16, 2026 12:31
@n0rpan
n0rpan marked this pull request as ready for review November 16, 2026 12:58
@n0rpan

n0rpan commented Nov 16, 2026

Copy link
Copy Markdown
Collaborator Author

No P1–P3 findings at this head. Read the complete facts implementation in prior passes and every change since: the facts/storage code remains unchanged from my last MERGE-OK. The added synthetic gateway fixture correction uses a process-local atomic sequence plus exclusive mkdir/retry, eliminating timestamp collisions without sharing fixture directories or weakening isolation. Journal durability/recovery, bounded history and pending checkpoint replay retain the previously reviewed behavior. No builds or credential reads; named gate green.

oxa-work still requires the orchestrator own independent read after mine.

@n0rpan

n0rpan commented Nov 16, 2026

Copy link
Copy Markdown
Collaborator Author

No P1–P3 findings. Read the current patch relative to the previously reviewed facts-storage patch: storage/situation logic is unchanged; fixture changes add the required preparing step to task-start observations, with other differences from the updated base. Checked bounded persistence/replay, checkpoint deduplication, trimming/gap reporting and durable append/directory-sync controls. Local acceptance, external acceptance and actual merge evidence remain separate. No builds/tests run locally. The orchestrator still needs its own oxa-work read.

@n0rpan
n0rpan marked this pull request as ready for review November 16, 2026 15:41
@n0rpan

n0rpan commented Nov 16, 2026

Copy link
Copy Markdown
Collaborator Author

No P1–P3 findings. Read the complete resolution delta against ab952f4. Facts persistence/replay, trimming, checkpoint semantics, situation projection and tests are unchanged; protocol context/lockfile compose the new base and the redundant fixture-root modification drops out. Checked that TaskStarted/TaskTransition and explicit merge evidence retain the previously reviewed semantics. No builds/tests run locally. The orchestrator still needs its own oxa-work read; no approval is implied.

@n0rpan
n0rpan marked this pull request as draft November 16, 2026 16:44
@n0rpan
n0rpan marked this pull request as ready for review November 16, 2026 16:47
@n0rpan n0rpan added the after-0.2.1 Approved work held until v0.2.1 is branched; merges in one batch after label Nov 17, 2026
@n0rpan n0rpan added this to the v0.2.2 milestone Nov 17, 2026
@oxa-agent

oxa-agent Bot commented Nov 17, 2026

Copy link
Copy Markdown
Contributor

O411 primary disposition handoff; existing sole writer retained.

SPLIT:1704 own lines. Existing worker-contract/facts writer remains sole writer; lead will only consume the public API. Extract existing protected facts_io212-line source with focused security tests, then projection/incidents and journal/replay slices <=400 each, maintaining atomic checkpoint/cursor semantics and existing public Facts::situation contract. Proto removals inspected: Running/Incident record declarations are extended, not dropped. Do not copy facts modules into lead or split away required crash invariants.

No branch edit, closure, archive or repeat full review by this disposition owner. Release #763 and security O413 remain first.

@oxa-agent

oxa-agent Bot commented Nov 17, 2026

Copy link
Copy Markdown
Contributor

O417/O418 supersedes the size-only SPLIT request in my previous O411 handoff. Stop forced size splits; retain current sole writer and any split work already made. Existing reported checks are green; use existing review evidence for a gate re-verdict and normal ready/required CI where still draft. No new full third lane review. Grandfathered size alone is not a blocker. Real findings, parent dependencies, owner-read and authority protections remain; READY here is author readiness, not merge authorization. No branch change or merge by this disposition owner.

@oxa-agent

oxa-agent Bot commented Nov 17, 2026

Copy link
Copy Markdown
Contributor

@codex security review

@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review

Security review completed. No security issues were found in this pull request.

Reviewed commit: 4cf042b2af

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

@oxa-agent oxa-agent Bot removed the after-0.2.1 Approved work held until v0.2.1 is branched; merges in one batch after label Nov 17, 2026
@oxa-agent

oxa-agent Bot commented Nov 17, 2026

Copy link
Copy Markdown
Contributor

READY FOR OWNER READ — exact head 4cf042b. Current GitHub state: ready, MERGEABLE, no failed or pending checks. Codex code summary 6915322560 reports no findings/unresolved findings at this head; canonical security receipt 6949516423 explicitly reports no security issues at 4cf042b. No repeat review or head refresh is needed. Narrow read scope: protected facts-root placement outside worker-writable binds, no-follow/ownership checks, journal reservation/recovery/ack ordering and durable directory sync. This is library-only: no CLI activation/store/thread on the normal path; preserve default-off orchestrator activation. Read #684 before dependent #759. No merge performed.

@oxa-orchestrator
oxa-orchestrator Bot added this pull request to the merge queue Nov 17, 2026
Merged via the queue into main with commit b0c21ab Nov 17, 2026
90 checks passed
@oxa-orchestrator
oxa-orchestrator Bot deleted the agent/629-lead-facts branch November 17, 2026 15:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant